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1.   Introduction 

For  the  past  several  years,  the  Department  of  Computer  Science 
of  the  University  of  Illinois  at  Urb ana-Champaign  has  been  developing 
computer  software  for  the  PLATO  Computer-Aided- Instruction  (CAI)  system 
(Alpert  and  Bitzer,  1970)  to  aid  in  teaching  introductory  computer  science 
courses  (Nievergelt,  1975).   This  author  has  been  responsible  for  designing 
and  coordinating  the  development  of  a  series  of  PLATO  lessons  to  teach 
the  computer  programming  language  FORTRAN.   The  series  consists  of  about 
15  lessons  (only  12  of  which  were  completed  in  time  for  use  in  this  experiment, 
see  Table  l)  and  a  student  should  spend  an  average  of  1+0-50  minutes  using 
each  lesson.   The  lessons  consist  of  displays,  diagrams,  textual  material, 
and  quizzes,  and  the  students  interact  with  the  lessons  to  learn  the  material 
and  are  typically  quizzed  over  it  during  the  lesson,  so  they  can  check  out 
their  understanding. 

The  lessons  were  initially  written  mainly  by  students  for 
class  and  independent  study  projects,  and  were  revised  and  improved  by 
members  of  the  staff.   Thus  the  lessons  do  not  follow  a  particular 
teaching  method  or  style,  but  each  covers  the  assigned  material  in  its 
own  way.   Examples  of  displays  from  lessons  are  presented  in  Figures  1  and  2. 

Figure  1  is  taken  from  lesson  fortintro,  which  students  took 
during  the  first  week  of  class.   In  it,  the  student  gets  a  problem,  and  he 
constructs  a  flowchart  using  natural  English  (actually  the  small  but 
usually  sufficient  subset  which  the  program  understands),  with  the 


The  author  would  especially  like  to  thank  Dr.  H.  G.  Friedman,  Sandra  Leach, 
and  Jeffrey  Barber  for  their  invaluable  help  in  this  area. 


Week 


Lesson 

Name 


Table  1 
PLATO  Lessons  Used 

Author ( s ) 


Brief  Description 


1 
1 
2 

3 

k 


csintro      P.  Kimble        Introduction  to  computers 
fortintro    R.  Montanelli    Introduction  to  programming 
fortarith    J.  Barber       FORTRAN  arithmetic 
fortfmtl     G.  Strass        Simple  FORTRAN  print  formats 
fortarrayl   H.  G.  Friedman   One  dimensional  FORTRAN  arrays 
K.  Levin,  J.  Fill 

FORTRAN  IF  statements 

FORTRAN  DO  loops 

Sorting  algorithms 

Two  dimensional  FORTRAN  arrays 

Advanced  FORTRAN  formatting 


5 

fort if 

G. 

Strass 

7 

fort do 

B. 

Barta 

8 

sorting 

J. 

Cupec 

9 

fortarray2 

L. 

Kawell 

10 

fortfmt2 

D. 

Dodgson, 

A. 

Papamarcos , 

N. 

Klein 

11 

fortsubl 

K. 

Slaughter 

13 

fort char 

V. 

Guarna 

FORTRAN  subroutines 
FORTRAN  character  handling 


Figure  1 
PLATO  Display  from  Lesson  fortintro 


Print  title 


Print  column  headings 


Set  number  to  1 


What  should  the  computer  do  next? 


» 


Calculate  number  squared 


Calculate  number  cubed 


Press  (help)  for  a  general  hint. 
Use  lEPfiSEj  to  change  an  answer. 
Press  fc'ftTft)  to  see  computer  printout 
Press  (me xt]  to  check  your  answer. 


computer's  help.   The  problem  is  to  create  a  table  of  the  numbers 
from  1  to  25  inclusive,  with  their  squares  and  cubes.   Students  are  first 
shown  a  description  of  the  problem  and  a  copy  of  the  computer  output. 
Then  they  try  to  answer  the  questions  to  create  the  flowchart.   The 
answer  to  the  question  in  Figure  1  is  "Print  number,  square,  and  cube", 
but  the  program  can  accept  a  reasonable  range  of  correct  answers,  like 
"The  computer  should  output  the  number  and  its  square  and  cube.".   This 
is  possible  mainly  through  the  use  of  synonyms  and  ignorable  words.   Some 
incorrect  answers  like  "Print  number"  are  answered  by  a  suitable  response 
"The  computer  prints  an  entire  line  at  a  time.   Something  else  should  be 


on  the  line,  also.   Press  ERASE  and  try  again."  After  each  incorrect 
answer  whether  understood  or  not,  the  student  gets  a  more  detailed  hint, 
until  finally  the  answer  is  given. 

Figure  2  shows  a  drill  from  the  2-dimensional  array  lesson 
fortarray2.   After  some  introduction  to  2-dimensional  arrays,  the 
student  is  expected  to  enter  the  appropriate  subscript  at  the  arrow 
(in  cell  3,3  in  the  figure)  which  is  flashed  on  and  off  to  get  the 
student's  attention.   Each  student  responds  to  five  (or  more  if  he 
wants)  of  these  subscript  pairs. 

In  order  to  study  the  effectiveness  of  these  lessons  in  teaching 
FORTRAN,  an  experiment  was  designed,  using  CS  103,  an  introductory  computer 
programming  class  for  students  in  the  behavioral  and  social  sciences. 
Since  about  1/2  of  the  class  time  is  spent  on  FORTRAN,  it  seemed  ideally 
suited  for  the  experiment. 


Figure  2 
PLATO  Display  from  Lesson  fortarrayl 


To  set  up  this  table  we  have  used  the  following 

statement:     •   REAL  RAT  (8,  4) 

Imagine  that  each  block  in  the  table  represents 

a  variable  in  this  array.  (The  name  of  the  variable 

is  at  the  top  of  each  block.) 


What  integer  should 
be  inserted  at  the 
arrow  ? 


Examp 1 e  1 


WEIGHT 

AGE 

SEX 

LENGTH 

RAT(    , 

) 

RATC    , 

) 

RATC    , 

) 

RATC    ,    ) 

10.6 

50. 

1 . 

11.7 

RATC    , 

) 

RATC    , 

) 

RAf  C    , 

) 

RATC    ,    ) 

14.3 

48. 

2. 

9.  4 

RATC   , 

9.3 

) 

RATC    , 
45. 

) 

RATC    , 

T 

2. 

) 

RATC    ,    ) 
12.3 

RAT(    , 

) 

RATC    , 

). 

RATC    , 

) 

RATC    ,    ) 

11.5 

52. 

1 . 

10.5 

RATC    , 

) 

RATC    , 

) 

RATC    , 

) 

RATC    ,    ) 

12.0 

55. 

2. 

11.8 

RAT(    , 

) 

RATC    , 

) 

FrlTC    , 

) 

RATC    ,    ) 

13.2 

60. 

1  . 

10.5 

RATC    , 

) 

RATC    , 

) 

RATC    , 

) 

RATC    ,    ) 

15.8 

49. 

1 . 

12.2 

RATC    , 

) 

RAT  C    , 

) 

RATC    , 

) 

RATC    ,    ) 

11.7 

51. 

-**- 

i. . 

10.  9 

2.   Method 

CS  103  is  ordinarily  taught  with  2  lectures  and  1  discussion 
meeting  per  week.   The  lectures  bring  all  the  students  (56  in  this  case) 
together,  and  they  are  split  into  k   groups  of  10-20  students  each  for  discussions 
with  a  teaching  assistant  at  the  end  of  the  week.  For  this  experiment,  1/2 
of  the  students  were  randomly  selected  to  attend  a  one-hour  PLATO  session 
instead  of  one  of  the  ordinary  lectures.   This  was  done  by  assigning 
students  with  odd  computer  numbers  to  the  PLATO  (p)  group.   The  computer 
numbers  were  handed  out  as  students  entered  the  classroom  on  the  first 
day.   The  remaining  students  (the  non-PLATO  or  NP  group)  acted  as  a  control 
group  and  were  taught  in  the  traditional  manner,  without  PLATO. 

The  two  groups  of  students  were  kept  separate  for  the  entire 
semester,  using  the  following  procedures:   1)  Students  were  fully 
informed  about  the  nature  of  the  experiment  and  were  assured  that  if 
the  two  groups  demonstrated  significantly  different  levels  of  achievement,  the 
group  achieving  the  poorer  record  would  have  its  grades  raised  to  match  those 
of  the  better  group.   2)  The  PLATO  terminals  were  reserved  for  9:00  am. 
Wednesday  morning,  the  same  time  as  the  lecture  met.   Thus,  it  was 
impossible  for  a  student  to  attend  the  lecture  and  the  scheduled 
PLATO  session.   Although  a  student  could  try  to  take  his  PLATO  lesson 
at  some  other  time,  due  to  the  heavy  scheduling  of  terminals  and  the 
shortage  of  ECS  (computer  secondary  memory),  it  was  very  difficult  for 
a  student  to  take  a  lesson  at  some  other  time.   3)  The  lessons  were 
restricted  so  that  no  one  except  students  in  CS  103  and  a  few  people  on 
our  staff  could  execute  them. 


Throughout  the  semester,  the  students  were  compared  on  various 
measures  of  achievement  (computer  programs,  two  hour  exams,  and  the  final 
exam)  and  on  interest  and  satisfaction  questionnaires. 


3.   Results 

a.  Questionnaires  On  the  first  day  of  class,  the  students  completed 
two  questionnaires  (biographical  data  and  attitude  toward  computers,  see 
appendices  A  and  B).   In  order  to  make  sure  that  the  two  groups  did  not 

differ  on  any  relevant  variables,  they  were  compared  on  the  following 

2 
variables  from  the  above  questionnaires,  using  t-tests  or  x  s  where 

appropriate:   age,  sex,  year  in  school,  grade  point  average,  number  of 

hours  of  college  math,  number  of  hours  of  statistics,  computer  experience, 

and  various  attitudes  toward  computers.   None  of  these  variables  indicated 

a  significant  difference  between  the  two  groups. 

Six  weeks  into  the  semester,  after  the  first  hour  exam,  students 

were  asked  for  their  opinions  on  the  course  (appendix  C).   Forty-four 

students  responded  when  asked  if  they  would  like  to  switch  to  the  other 

group.   The  responses  were: 

don '  t  don '  t 
yes   no   know   care 

P 

NP 

Responses  of  the  non-PLATO  group  to  the  questions  about  their  smaller  lecture 
and  of  the  PLATO  group  about  the  PLATO  lessons  are  presented  in  Tables  2  and 
3.   To  summarize,  the  NP  group  thought  that  the  class  atmosphere  was  somewhat 
more  casual  and  that  they  could  get  questions  answered  more  easily  in  the 
smaller  lecture.   The  P  group  felt  that  the  lessons  were  enjoyable,  better 
than  lectures  on  the  same  topic,  and  that  they  learned  something  from  them. 
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3 

16 

2 

0 

3     , 

11 

k 

5 

Table  2 
Responses  of  the  NP  Group  to  Questions 
about  Comparing  Their  Smaller 
Lecture  to  the  Large  One 


A.  Keeping  up  with  the  lecture  is: 

easier  (2)        no  difference  (21) 

B.  Getting  questions  answered  is: 

much  easier  (l)     easier  (8)    no  difference  (1^) 

C.  The  atmosphere  of  the  class  is: 

more  casual  (9)    the  same  (l^) 

D.  If  the  entire  class  were  together  you  would  learn: 

the  same  (17)      less  (6) 

In  the  above  table,  only  those  responses  used  are  listed,  and  the 
number  of  students  giving  each  response  is  given  in  parentheses. 


Table  3 
Student  Reactions  to  PLATO  Lessons 


A.   Compare  the  lesson  with  how  a  lecture  might  teach  the  same  topic. 


The  lesson  was  a.  much 

better 

1. 

csintro 

2 

2 

algolingo* 

5 

fortintro 

k 

k. 

fortarith 

5 

5. 

fortfmtl 

2 

6 

fortarrayl 

2 

7. 

fort if 

2 

b.  better  c.  equal  d.  worse  e.  much 

worse 


7 
2 

9 
7 
6 

2 

1+ 


6 
3 
3 
3 
7 
11 

7 


2 
3 

2 
2 
3 
3 
3 


omits 

1+ 

13 

3 

k 

3 
3 
3 


B.  How  helpful  was  the  lesson  for  the  exam? 

a.  extremely  b.  very  c.  somewhat  d.  not  at  e.  harmful 

all 


1 

csintro 

1 

1+ 

10 

3 

2 

algolingo* 

3 

5 

3 

fortintro 

2 

3 

12 

1 

k 

fortarith 

3 

9 

5 

5 

fortfmtl 

1 

7 

9 

6 

fortarrayl 

2 

2 

12 

2 

7 

fortif 

1 

2 

11 

2 

omits 

3 

13 

3 

3 

k 


Table  3  (cont'd) 


C.  How  much  did  you  learn  from  the  lesson  (whether  the  material  was  on  the 
exam  or  not )  ? 


i 

i.  very 
much 

b .  quite 
a  bit 

c.  some 

d.  very 
little 

e .  nothing 

1 

csintro 

1 

9 

6 

2 

2 

algolingo* 

1 

3 

3 

1 

3 

fortintro 

1 

11 

5 

1 

k 

fortarith 

6 

7 

if 

5 

fort  flat  1 

3 

6 

8 

6 

fortarrayl 

3 

8 

5 

1 

1 

7 

fort  if 

6 

11 

1 

omit 

3 

13 

3 

k 
3 

3 


D.   How  much  did  you  enjoy  the  lesson? 


a.  very 
much 

b .  quite 
a  bit 

1 

csintro 

k 

9 

2 

algolingo* 

3 

2 

3 

fortintro 

h 

11 

4 

fortarith 

6 

8 

5 

fortfmtl 

5 

7 

6 

fortarrayl 

5 

7 

7 

fort if 

3 

9 

c.  very   d.  not     e.  hated 
little    at  all      it 

5 

3 

3 

3 

5 

4        11 

4        11 


omit 

3 

13 

3 

3 
3 


*  optional  lesson 
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b.  Drop  rates  Although  56  students  actually  attended  at  least 
one  class  session  and  thus  were  assigned  to  a  group,  only  ^7  remained 
at  the  end  of  the  semester.   Of  the  other  nine  students  (see  Table  h) 
two  were  never  actually  enrolled  and  three  others  never  even  turned  in 
the  first  homework  problem  which  was  due  the  second  week  of  class  and  was 
basically  only  a  keypunching  exercise.   Thus  of  51  students  who  actually  began 
the  course,  only  four  dropped.   The  comparable  figures  for  fall  1973  were 
"that  73  students  began  the  course  and  12  dropped.   These  two  sets  of 
figures  are  not  significantly  different,  and  any  trend  toward  fewer 
drops  in  fall  197^  is  undoubtably  due  to  the  fact  that  the  experiment  itself 
helped  maintain  student  interest. 

Comparisons  between  the  two  groups  in  fall  197^  are  equally  non- 
significant.  Of  the  four  students  who  dropped  after  doing  some  work,  two 
were  from  each  group.   Although  four  of  the  five  other  students  were 
in  the  NP  group,  it  would  be  extremely  difficult  to  attribute  these  losses 
to  anything  other  than  chance. 


Table  k 
Analysis  of  Drops 


Number 

of 

Student 

;s 

Group 

1 

NP 

1 

NP 

3 

IP,  2NP 

3 

2P,  INP 

1 

NP 

met  one  class,  never  enrolled 

auditor,  did  no  work 

did  no  work 

dropped  after  first  exam  (6  weeks) 

dropped  after  second  exam  (11  weeks) 
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c.  Achievement  Grades  in  CS  103  are  based  on  four  components: 
machine  problems  (computer  programs),  two  hour  exams,  and  a  final  exam. 
Means  and  standard  deviations  for  the  two  groups  on  these  four  variables 
are  presented  in  Table  5-   Although  the  NP  group  did  better  on  all  three 
exams,  the  multivariate  test  for  differences  between  the  two  groups  was  not 
significant  (F  =  1.5^  with  h   and  39  degrees  of  freedom).   This  test  was 
based  on  the  hk   students  who  took  all  three  exams  (Of  the  ^7  students  who 
finished  the  course,  two  skipped  the  final  exam  and  failed  the  course, 
while  one  had  an  excuse  from  the  second  hour  exam.). 

Table  5 
Means  and  Standard  Deviations  of  the 
Two  Groups  on  the  Achievement  Variables 


Machine 

Hour 

Hour 

Final 

Problems 

Exam  1 

Exam  2 

Exam 

PLATO 

means 

152 

67 

5^ 

lM 

SDs 

28.3 

15.9 

18.0 

33-0 

non- PLATO 

means 

152 

!k 

67 

153 

SDs 

23-9 

13.^ 

17.1 

26.0 

Univariate  analyses  of  variance  for  these  same  four  variables  are 

presented  in  Table  6,  for  two  reasons:   First,  to  include  each  student  who 

took  any  particular  exam  (or  worked  on  several  machine  problems)  in  the 

analysis  for  that  variable;  and  secondly  to  point  out  the  near  significant 

results  (at  the  .05  level)  on  the  two  hour  exams.   Although  these  results 

do  not  show  significant  differences  between  the  two  groups  in  knowledge 

gained  in  the  course,  they  indicate  a  possible  problem  which  should  be 

watched  in  the  future. 
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Table  6 
Univariate  Analyses  of  Variance 
on  the  Achievement  Variables 


Mean 

Degrees 

of  Freedom 

Variable 

square 

num 

denom 

F 

p 

Machine  problems 

51 

1 

i+9 

.03 

.85 

Hour  exam  1 

799 

1 

1+9 

3.89 

.05^ 

Hour  exam  2 

1527 

1 

^5 

3.81 

•  057 

Final  exam 

2195 

1 

^3 

2.33 

.135 

It  should  also  be  mentioned  here,  that  after  the  first  exam, 
the  author  speculated  that  the  fact  that  he  made  up  the  exam  might  have 
had  some  effect  on  the  results.   So  for  the  second  exam,  two  other 
professors  in  the  department  made  up  the  exam  from  the  textbook,  using 
the  course  syllabus.   Since  this  procedure  had  no  noticeable  effect  on 
the  results,  it  was  abandoned  for  the  final  exam. 

d.  PLATO  usage  Students  in  the  PLATO  group  entered  k3   separate 
lessons  (programs)  on  PLATO,  during  the  course  of  the  semester.   The  lessons 
could  be  roughly  classified  as  required  (12  of  these,  see  Table  l),  optional 
(7,  FORTRAN  or  FORTRAN-related) ,  system  (k,   routing  and  messages),  and  others 
(20  computer  science  lessons  unrelated  to  CS  103).   Some  PLATO  usage  statistics, 
divided  according  to  these  four  categories,  are  presented  in  Table  7«   The 
most  interesting  statistics  are  that  on  the  average,  students  entered  9»9 
of  the  12  required  lessons  and  spent  184.5  minutes  in  required  lessons. 
Although  the  first  figure  is  accurate,  the  second  is  actually  a  minimum, 
due  to  the  following  three  problems.   First,  time  was  not  properly  recorded 
for  lesson  sorting,  which  showed  89  uses  and  only  a  total  of  five  minutes 
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spent  in  it,  and  for  some  other  lessons  which  were  not  part  of  the  course. 
Second,  if  the  system  went  down  or  in  some  cases  if  the  student  quit  in  the 
middle  of  a  lesson,  no  time  was  recorded.   Finally,  due  to  an  unknown  error, 
in  some  cases  no  time  was  recorded  in  a  lesson  even  though  the  student 
entered  the  lesson  one  or  more  times.   Thus,  while  the  average  student 
spent  about  10  hours  at  the  terminal  (this  data  recorded  independently  by 
the  PLATO  system),  only  about  3  1/2  hours  of  that  was  properly  assigned 
to  individual  lessons.   A  table  of  these  statistics  by  individual  lesson 
is  presented  in  Appendix  D. 

In  order  to  discover  the  relationship  between  time  spent  on  the 
required  PLATO  lessons  and  course  grades,  these  two  variables  were  correlated. 
The  resulting  correlation  of  .58  which  accounts  for  about  3M&  of  the  variance 
and  is  significant  at  the  .005  level  indicated  a  strong,  positive  relationship 
between  time  spent  on  the  relevant  PLATO  lessons  and  course  grade.   (The 
correlation  with  total  time  from  all  lessons  was  .kh   and  with  total  time 
on  PLATO  was  .18.)   In  order  to  discover  if  this  correlation  might  be  an 
artifact  due  to  brighter  students  working  harder,  the  correlation  was 
recomputed,  partialling  out  GPA.   The  resulting  partial  correlation  of  .55 
indicated  that  the  relationship  between  hours  on  PLATO  and  course  grade  was 
not  due  to  students  with  higher  GPAs  working  harder  on  PLATO. 
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Table  7 

Summary  of  PLATO  Lesson 

Usage  per  Student 


Lesson  type 
required     optional    system    other    TOTALS 


Number  of 
times  entering 

Time  spent  (in 
minutes) 

Number  of 
lessons 


32.6 


184.5 


9-9 


3.2 


6.1 


1.5 


15.2 


12.6 


2.6 


2.0 


1.0 


53.0 


k.G  207.8 
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e.  Course  evaluation  questionnaire  The  Computer  Science  Department 
routinely  administers  a  standardized  course  evaluation  questionnaire 
(Appendix  E)  in  each  of  its  courses,  near  the  end  of  the  semester.   Although 
the  questionnaires  are  administered  anonymously  and  thus  could  not  be 
separated  into  PLATO  and  non- PLATO  groups,  it  was  thought  that  some 
comparisons  might  be  made  with  results  from  the  previous  year  (fall,  1973) > 
and  that  some  comments  about  PLATO  might  be  made  in  response  to  the  open- 
ended  questions  on  the  back  of  the  questionnaire. 

The  h-2   objective  questions  were  scored  on  a  h,   3>  2,  1  scale 
(for  responses  A,  B,  C,  and  D) ,  and  analyses  of  variance  were  performed  for 
each  question  between  the  two  semesters.   Results  are  in  appendix  E.   Even 
if  one  tried  to  interpret  all  of  the  significant  results  (which  would  be  a 
very  questionable  statistical  procedure,  due  to  the  correlations  between  the 
questions,  and  the  pure  probabilities  which  state  that  about  2  of  h-2   such 
tests  would  be  significant  at  the  .05  level  by  chance),  nothing  of  interest 


15 


is  found.  Most  of  the  results  appear  to  be  unrelated  to  the  experiment, 
such  as  the  1973  group  reporting  that  the  instructor's  voice  was  more 
animated  and  that  he  varied  the  pace  of  lecturing  more  than  the  197^ 
lecturer  or  the  197^  group  saying  that  grading  of  machine  problems  was 
more  fair  or  that  TAs  were  better  supervised.   (The  author  was  the 
instructor  in  both  cases.)   Others  were  related,  but  unimportant,  (i.e. 
The  1973  lectures  were  more  related  to  the  reading  assignments,  because 
the  PLATO  lessons  essentially  replaced  the  lectures  which  would  have 
gone  over  the  reading  material.) 

Of  the  29  students  who  completed  the  questionnaire,  6  mentioned 
PLATO  on  the  back.  While  one  of  these  was  from  a  student  who  had  wanted 
to  get  a  chance  to  use  PLATO  (presumably  from  the  NP  group!),  the  other 
five  were  critical  to  varying  degrees.   A  couple  made  general  comments 
like  "it  was  frustrating"  or  the  lessons  were  "too  philosophical",  while 
two  others  suggested  that  PLATO  should  be  eliminated  or  improved. 

These  sentiments  were  basically  repeated  in  responses  to  the 
question  "How  woiild  you  improve  the  course"  which  was  part  of  a  short 
questionnaire  attached  to  the  final  exam.   Additionally,  three  students 
mentioned  that  they  needed  more  time  on  PLATO. 
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U.   Discussion 

There  were  three  interesting  results  of  the  study.   First, 
although  students  seemed  reasonably  satisfied  with  PLATO  early  in  the 
semester,  the  comments  made  at  the  semester's  end  indicated  some 
dissatisfaction.   Four  possible  explanations  are:   l)  early  in  the 
semester  students  find  PLATO  new  and  interesting,  but  the  novelty 
wears  off  by  the  semster's  end,  2)  students  are  more  concerned  with 
grades  at  the  end  of  the  semester,  3)  earlier  lessons  were  in  better 
shape  than  later  ones,  and  k)   ECS  problems  were  worse  later  in  the 
semester.   Undoubtably  each  of  these  explanations  played  some  part 
in  student  attitudes,  and  of  course  little  can  be  done  to  change  the 
effect  of  the  first  two.   However,  it  was  the  case  that  some  of  the 
earlier  lessons  were  better  tested  than  some  of  the  later  ones.   Thus 
some  additional  improvements  for  these  lessons  might  be  indicated. 
Finally,  the  shortage  of  ECS  could  have  had  a  larger  effect  at  the  end 
of  the  end  of  the  semester,  because  as  students  fall  behind  or  need  to 
review,  they  create  a  demand  for  many  lessons  at  the  same  time,  and 
that  wasn't  possible.   However,  added  ECS  in  January,  1975  removed  this 
problem. 

The  most  encouraging  result  was  the  strong  relationship  between 
the  amount  of  time  spent  in  the  required  PLATO  lessons  and  the  course 
grade.   If  this  is  a  cause  and  effect  relationship,  it  illustrates  the 
usefulness  of  the  lessons. 

The  final  interesting  result  was  the  lack  of  a  significant 
difference  between  the  two  groups  on  achievement  variables.   With 
some  of  the  problems  encountered,  this  result  was  good,  in  spite  of  the 
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fact  that  the  observed  differences  favored  the  NP  group. 

The  above  results  suggest  several  improvements  in  the  course. 
First,  one  of  the  benefits  of  CAI  is  its  self -pacing,  self- scheduling 
feature,  and  this  was  basically  absent  due  to  the  lack  of  ECS,  but  this 
problem  has  now  been  remedied.   Secondly,  efforts  are  being  made  to 
improve  the  quality  of  many  of  the  lessons.   These  mainly  involve 
increased  use  of  animation,  simulation,  exercises,  and  other  forms  of 
interactive  material  to  help  maintain  student  interest  and  allow 
students  to  practice  what  they  have  learned. 

An  example  of  a  new  exercise  appears  in  Figure  3-   For  this 
experiment,  the  introductory  formatting  lesson  fortfmtl  consisted  of 
a  lot  of  text  with  a  fairly  small  number  of  exercises  for  the  student. 
In  response  to  student  comments  that  the  lesson  was  dull  and  to  complaints 
that  it  didn't  give  students  a  chance  to  practice,  three  of  these  exercises 
were  added  (for  I,  F,  and  E  formats)  and  a  simulator  is  being  written. 
The  problems  for  the  exercise  are  generated  randomly  (within  some 
constraints),  and  students  type  in  answers  in  the  boxes  until  they 
achieve  the  criterion  set  by  the  instructor  (5  correct  in  this  example). 
The  computer  keeps  track  of  students'  results,  and  lets  them  procede 
when  the  criterion  is  met. 

A  third  possible  mechanism  for  improving  student  performance 
would  be  through  encouraging  or  forcing  the  students  to  take  the  PLATO 
lessons.   In  CS  103,  students  were  told  to  use  the  lessons,  but  apparently 
some  did  not,  although  there  were  some  problems  with  the  timing  data. 
In  any  case,  the  CAISMS  project  (Anderson  et  al,  197^)  using  computer-managed 
instruction  showed  that  students  performed  better  if  they  were  forced 
to  study.   Also,  the  correlation  between  course  grade  and  time  in  the 
required  lessons  in  this  study  indicated  that  using  the  lessons  improved 
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Figure  3 
PLATO  Display  from  Lesson  fortfmtl 


PAGE  13. 


EXERCISES    IN    I    FORMATS 


Below  you  will  be  given  an  INTEGER  and  a  FORMRT 
statement  and  then  you  are  to  show  how 
that  number  would  be  PRINTed.  In  this 
exercise  spacing  is  important  and  wi/11  be 
checked  in ' determining  whether  or  not  your 
answer  is  correct.  You  must  get  5  problems 
correct  to  continue  with  this  lesson. 

To  indicate  that  a  number  is  too  large  for  the 
format  use  *'s  to  fill  in  the  field. 


INTEGER  NUMBER  -♦  -28  2 
FORMAT  SPECIFICATION  -♦  15 
YOUR  ANSWER   fc 


You  now  have  4  of  4  correct . 


Press  -STOP-  to  continue  with  the  lesson 
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grades.   Perhaps  some  small  amount  of  credit  could  be  given  to  students 
who  complete  lessons,  in  order  to  encourage  them  to  do  so.   It  should  be 
noted  that  the  effect  of  PLATO  on  CS  103  (or  perhaps  any  course)  may  be 
greater  than  the  simple  substitution  of  50  minutes  of  CAI  material  for 
a  50  minute  lecture.   Students  may  be  more  likely  to  skip  a  PLATO  lesson 
than  a  lecture  because  they  could  do  it  later  or  because  there  is  no 
instructor  to  notice  their  absence.   Additional  work  should  consider 
student  motivation  and  attendance  at  PLATO  and  in  lectures. 

In  summary,  CAI  materials  can  replace  some  lectures  on  the 
FORTRAN  language  in  an  introductory  computer  science  course.   However,  more 
effort  must  be  spent  on  lesson  development  and  evaluation  than  was  originally 
suspected. 
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APPENDIX  A 


QUESTIONNAIRE 
Computer  Science  103 

USER  number  Signature  Soc.  Sec.  No. 

(h   digits  from  upper 
right  of  envelope) 

Attitude  toward  Computers 

Please  check  your  degree  of  agreement  or  disagreement  with  each  of  the  fol- 
lowing statements: 

STRONGLY  STRONGLY 

AGREE      AGREE     DISAGREE      DISAGREE 

Computers  are  infallible  

Computers  are  overrated  

Computers  dehumanize  institutions     

Computers  are  useful  to  society       

Computers  are  often  misused  ■  


Their  advantages  outweigh  their 

disadvantages  

Please  check  your  familiarity  with  PLATO: 
never  heard  of  it     heard  about  it     tried  it  once  or  twice    used  it  in  a  class 


have  programmed  it 


Familiarity  with  computers  other  than  PLATO: 

ran  a  program  or  2   know  a  little  FORTRAN    had  a  course  or 
never  used  one    (SOUPAC,  for  ex.)    or  some  other  language   considerable  experien 
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APPENDIX  B 

CS  103  Biographical  Questionnaire  Fall  197I+ 

In  order  to  make  some  of  the  later  machine  problems  more  interesting, 
the  data  obtained  from  this  class,  via  this  questionnaire,  will  be  used.   You 
will  be  given  copies  of  this  data  and  asked  to  compute  some  statistics  (all  of 
which  will  be  explained  in  class)  on  it.   Additionally,  this  information  will 
be  used  to  examine  the  degree  of  variation  in  the  backgrounds  of  the  students, 
so  that  the  material  presented  will  be  at  the  appropriate  level. 

Please  mark  you  answers  in  the  space  on  the  right. 

1.  Social  Security  number  .  1  ■  1  1  t   ,  .  ,  , 

1  9 

2.  Sex  (M  or  F) 

3.  Departmental  abbreviation  (eg.  PSYCH,  SOC,  or  NONE 

if  unassigned) 

h.       Age  in  years 


10 


• I L—l I I I 

11  16 


< t I 

17  18 

5.  Year  in  school  (1  for  Freshman,  2  for  Soph.,  3  for 

Jr.,  k   for  Sr.,  5  for  Grad. )  u_1 

19 

6.  Height  (feet  and  inches)  t ,  ft.  ,  ,  ,  in. 


<-j> 


7.  Weight  (pounds) 

8.  Approximate  college  GPA 

9.  Number  of  hours  of  college  mathematics 

10.  Number  of  hours  of  statistics  (any  departments) 


L_J L_J 

25    27 

I — UJ — 1 — 1 

29     32 

33^ 
35~36 


38 


11.  Computer  experience  0  none  

1  a  little  37 

2  used  SOUPAC 

3  some  knowledge  of  a  language 
k  a  lot  (Why  are  you  in  CS  103?) 

12.  Expected  grade  in  CS  103  (A,  B,  C,  D,  or  E) 

13.  Place  a  1  if  you  would  object  to  the  data  on  this 

questionnaire  being  made  available  to  the  other 

students  in  this  class  •  t , 

40 

Ik.        List  the  numbers  of  those  questions  whose  responses  you  wish  kept  confidential. 


15-   Place  a  1  if  you  would  object  to  the  data  be  inn,  made  available  to  the  class, 
anonymously  (without  soc.  sec.  nos.). 
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APPENDIX  C 
10/7  M  NAME 


Computer  Science  103  Questionnaire 

This  questionnaire  is  an  effort  to  get  your  opinions  about  the  experiment  being 
conducted  in  the  class.  Please  take  the  time  to  carefully  answer  all  questions 
which  pertain  to  your  half  of  the  experiment.  Your  comments  will  have  no  effect  on 
your  grade,  so  please  be  candid  and  as  critical  as  you  wish. 

I.   EVERYBODY 

A.  Would  you  rather  switch  to  the  other  section:  Yes   no   don't  know 

don't  care 

B.  Below,  and/or  on  the  back,  please  list  any  comments,  suggestions,  and 

criticisms  which  you  have  of  CS  103  this  semester,  especially  any  which 
pertain  to  the  experimental  division  of  the  class  into  2  sections.  Say 
what  you  like  and  what  you  don't  like  about  the  course. 


C.  List  any  exam  questions  which  you  feel  you  were  unprepared  for,  and  say  why. 


D.  Would  you  prefer  to  have  the  next  exam  in  the  evening?  Yes   No 

E.  IF  (you're  in  the  PLATO  section)  GO  TO  IV 

II.  REDUCED  CLASS  SIZE  GROUP  -  In  answering  the  following  questions,  compare  the 
smaller  Wednesday  lectures  with  the  larger  Monday  lectures. 

A.  Keeping  up  with  the  lecture  on  Wednesdays  is 

a.  much  easier  b.  easier  c.  no  difference  d.  harder  e.  much  harder 

B.  Getting  questions  answered  on  Wednesdays  is 

a.  much  easier  b.  easier  c.  no  difference  d.  harder  e.  much  harder 

C.  The  atmosphere  in  the  class  on  Wednesdays  is 
a.  more  casual  b.  same  c.  less  casual 

D.  If  the  entire  class  were  together  on  Wednesdays,  how  much  would  you  learn 
a.  a  lot  more  b.  more  c.  the  same  d.  less  e.  much  less 

III.  STOP,  look  up,  yawn,  and  stretch  to  indicate  you're  finished. 
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APPENDIX  C  (cont'd) 

IV.  PLATO  GROUP  -  You  have  had  an  opportunity  to  study  the  following  7  PLATO  lessons 
so  far  this  semester: 

1.  csintro  2.  algolingo  3«  fortintro  k.    fortarith  5.  fortfmtl  6.  fortarrayl 
7.  fortif 

Answer  each  question  below,  for  each  of  the  7  lessons  you  have  used. 

A.  Compare  the  lesson  with  how  a  lecture  might  teach  the  same  topic. 

The  lesson  was  a.  much  better  b.  better  c.  equal  d.  worse  e.  much  worse 
1.  2.  3.  ^.  5-  6.  7.  

B.  How  helpful  was  the  lesson  for  the  exam? 

a.  extremely  b.  very  c.  somewhat  d.  not  at  all  e.  harmful 
1.  . ,  2.  3. ^.  5.  6.  7. 


C.  How  much  did  you  learn  from  the  lesson  (whether  the  material  was  on  the 
exam  or  not)? 

a.  very  much  b.  quite  a  bit  c.  some  d.  very  little  e.  nothing 
1.  2.  3.  b.  5.  6.  7.  

D.  How  much  did  you  enjoy  the  lesson? 

a.  very  much  b.  quite  a  bit  c.  very  little  d.  not  at  all  e.  hated  it 
1.  2.  3.  b. 5.  6.  7.  

E.  GO  TO  III 


25 


APPENDIX 

D 

PLATO  Us; 

ige 

, 

LESSON 

NUMBER  OF 

NUMBER  OF 

NUMBER  OF 

NUMBER  OF 

NAME 

TIMES  USED 

BAD  EXITS 

MINUTES 

STUDENTS 

csintro 

V>- 

7 

680 

23 

fortintro 

k9 

19 

277 

21 

fortarith' 

101 

29 

678 

26 

fortfmtl 

96 

53 

137 

23 

fortarrayl 

97 

>+9 

622 

27 

fort if 

1A5 

37 

570 

26 

fort do 

78 

1+0 

576 

26 

sorting 

89 

77 

5 

22 

fortarray2 

kQ 

2h 

276 

23 

fortfmt2 

^ 

37 

281 

18 

fort sub 1 

62 

16 

539 

20 

fortchar 

9 

5 

6k 

5 

fortfmt 

16 

J+ 

0 

4 

fort sub 

9 

2 

53 

3 

fortcomp 

37 

24 

65 

18 

fortsubex 

5 

0 

35 

2 

fortfunct 

If 

0 

0 

1 

algolingo 

6 

6 

0 

6 

flowchrt 

3 

3 

0 

3 

csguide 

217 

17 

212 

26 

cscomments 

20 

19 

1 

7 

cslessons 

161 

3^ 

116 

27 

26 


APPENDIX  D  (cont'd) 


csr outer 

18 

18 

0 

11 

fortxl 

5 

2 

0 

2 

darwinl 

2 

0 

0 

1 

numquad 

1 

0 

Ik 

1 

root lab 

1 

1 

0 

1 

machlang 

1 

0 

1 

1 

cs 

5 

5 

0 

2 

racetrack 

13 

5 

100 

2 

basicintro 

2 

0 

0 

2 

snobol 

2 

2 

0 

1 

somaga 

2 

0 

0 

1 

basicrefl 

2 

2 

0 

1 

graphix 

1 

1 

0 

1 

monte  carlo 

1 

1 

0 

1 

traffic sim 

k 

1 

0 

2 

mazesearch 

1 

0 

0 

1 

turing 

1 

1 

0 

1 

kaids 

1 

1 

0 

1 

cs scrap 

1 

1 

0 

1 

snobolcomp 

2 

0 

0 

1 

minie 

1 

1 

0 

1 

TOTAIS 

1418 

5kK 

5302 

393 

AVERAGES 

3.6 

1.4 

13-5 

Note:  First  twelve  lessons  were  required,  next  seven  were  optional, 
next  four  were  system  lessons,  and  last  twenty  were  any  others. 
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APPENDIX  E 


Items   1-2U    describe   various   specific   things   that 

A  LECTURER  MAY  DO.   BLACKEN  THE  LETTER  ON  EACH 
XXXXXXXXXXXXXXXXXXXXX  LINE  TO  INDICATE  HOW  FREQUENTLY  THE  LECTURER 
XXXXXXXXXXXXXXXXXXXXX  IN  THIS  COURSE  DID  EACH  OF  THEM. 

xxxxxxxxxxxxxxxxxxxxx  USE  THE  CODE  ON  THE  RIGHT 


YOU  CAN  HELP  PROMOTE  GOOD  INSTRUCTION  DY  GIVING 
YOUR  CONSIDERED  EVALUATION  OF  CERTAIN  ASPECTS  Or 
YOUR  INSTRUCTOR  AND  THIS  COURSE.   PLEASE  ANSWER 
EACH  QUESTION  INDEPENDENTLY  SO  AS  TO  GIVE  A 
CORRECT  ASSESSMENT  OF  WHAT  YOU  THINK  ARE  STRONG 
AND  WEAK  POINTS  OF  THE  COURSE.    IF  YOU  CHOOSE 
NOT  TO  ANSWER  A  QUESTION,  PLEASE  GIVE  YOUR 
REASONS  ON  THE  BACK. 


A  =  Almost  Always 
B  =  Often 
C  =  Occasionally 
D  =  Almost  Never 


XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 


xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 
xxxxxxx 


1 . 

2. 

3. 

u. 

5. 
6. 
7. 
8. 


12. 
13. 
1U. 
15. 
16. 


Acted  interested  in  the  material. 

Was  well  prepared. 

Acted  relaxed. 

Looked  at  the  class  while  speaking. 

Enunc i ated  well  . 

Lectures  seemed  to  go  smoothly  following  a  logical  sequence  of  thought. 
Used  relevant  examples. 

Explained  clearly  and  explanations  were  to  the  point. 
9.  Emphasized  important  points  by  raising  voice,  repeating,  etc. 
10.  Made  you  interested  in  the  material. 
1.  Lectured  at  the  ability  level  of  the  class. 

Lectures  were  related  to  the  reading  assignments. 
Gave  clear  explanations  of  abstract  ideas. 

Made  clear  the  objectives  for  each  lecture  or  series  of  lectures. 
Followed  an  outline. 

Stimulated  your  intellectual  curiosity. 
17.  Seemed  to  have  very  recent  information  on  the  subject. 
xxxxxxx   18.  Answers  to  questions  were  relevant. 
xxxxxxx   19.   Varied  pace  of  lecturing. 

xxxxxxx  20.  Presented  material  not  in  the  reading  assignments. 
xxxxxxx  21.   Voice  was  animated. 
xxxxxxx  22.  Used  humor  effectively. 

xxxxxxx  23.  Answered  all  questions  (or  admitted  didn't  know  the  answer). 
xxxxxxx  2U .  Encouraged  questions  during  the  lecture. 


xxxxxxxxxxxxxxxxxxxxx  The  remaining  questions  refer  to  other  aspects 


XXXXXXXXXXXXXXXXXXXXX 

xxxxxxxxxxxxxxxxxxxxx 

xxxxxxxxxxxxxxxxxxxxx 
xxxxxxxxxxxxxxxxxxxxx 


OF  THE  COURSE.   USE  THE  SCALE  ON  THE  RIGHT  TO 
EXPRESS  YOUR  DEGREE  OF  AGREEMENT  OR  DISAGREEMENT 
WITH  EACH  STATEMENT. 


XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 


XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 

XXX 


A  =  Strongly  Agree 

B  =  Agree 

C  =  Disagree 

D  =  Strongly  Disagree 

G  =  Not  Applicable 


25.  Atmosphere  of  the  class  was  cordial. 

26.  Lecturer  was  patient  with  students  who  did  not  understand  the  material. 

27.  Overall  effectiveness  of  the  instructor  was  good. 

28.  Instructor  should  be  recommended  to  students  taking  a  similar  course. 

29.  Prerequisites  stated  in  the  course  catalog  were  necessary. 

30.  Machine  problems  were  worthwhile. 

31.  Grading  of  machine  problems  was  fair. 

32.  Written  homework  was  helpful. 

33.  Grading  of  writte-  homework  was  fair. 

3U .  Examinations  tested  the  course  material. 

35.  Grading  of  examinations  was  fair. 

36.  Adequate  arrangements  were  made  when  the  professor  was  absent. 

37.  Audio-visual  teaching  aids  were  used  effectively. 

38.  Reading  material  covered  the  course  material. 

39.  t.a.s  were  well  supervised. 
U0.  The  textbook  was  good. 

Ul.  A  lot  was  learned  by  taking  this  course. 

U2.  The  course  was  enjoyable. 
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APPENDIX  E  (cont'd) 

please  answer  the  questions  and  make  comments  about  the  aspects  of  the  course  indicated  below. 
Compliments  of  good  aspects  and  criticisms  of  poor  ones  are  all  welcome.  Your  comments  will  help 
your  instructor  interpret  the  results  from  the  questions  on  the  other  side.   remember,  your  instructor 
will  not  see  your  completed  evaluation  until  after  final  grades  for  this  course  have  been  turned  in. 

1.  What  grade  do  you  expect  to  get  in  this  course? 


2.   Comment  on  the  amount  of  time  required  by  this  course  compared  with  the  credit  given. 


3.  Comment  on  homework  and  machine  problems  (number,  difficulty,  relevance,  grading,  contribution 
to  final  grade,  etc.) 


U.  Comment  on  exams  (length,  difficulty,  grading,  contribution  to  final  grade,  etc.) 


5.  Please  comment  on  course  content  and  relevance  to  your  area  of  study. 


6.  Name  your  principal  instructor   ,  TA 

What  are  your  general  comments  about  the  instructor(s)  in  this  course? 


7.  General  comments.  Are  you  satisfied  with  what  you  got  out  of  this  course? 

Was  it  a  worthwhile  educational  experience?  How  would  you  improve  the  course? 


8.  What  is  your  reaction  to  this  questionnaire?  What  other  questions  should  have  been  asked?   Any 
questions  you  did  not  like  or  could  not  understand?   Please  give  reasons. 
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APPENDIX  E  (cont'd) 
Computer  Science  Course  Evaluation 
Questionnaire  Results 


riable 

Means 
1973         1974 

df (denom. ) 

F 

P 

1 

3-57 

3-41 

71 

.87 

•35 

2 

3.76 

3-79 

72 

.09 

•77 

3 

3-07 

3.34 

71 

I.78 

.19 

4 

3.38 

3-21 

72 

•99 

•32 

5 

3.76 

3.72 

72 

.08 

.78 

6 

3.56 

3.4l 

72 

•  67 

.42 

7 

3.^9 

3.24 

72 

2.39 

.13 

8 

3.11 

3-00 

72 

•  34 

.56 

9 

3-00 

2.71 

70 

1.89 

.17 

10 

2.45 

2.28 

71 

.68 

.41 

11 

3.18 

3.10 

72 

.16 

.69 

12 

3-54 

3- 07 

71 

7.66 

.007 

13 

2.77 

2.48 

71 

2.05 

.16 

14 

3.36 

3.14 

72 

1.21 

.28 

15 

3.64 

3-59 

71 

.11 

.74 

16 

2.42 

2.28 

72 

•37 

•  54 

17 

3.29 

3.15 

70 

•50 

.48 

18 

3.36 

3.31 

72 

.07 

•79 

19 

2.36 

I.72 

71 

11.54 

.001 
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APPENDIX  E   (cont'd) 


20 

2.18 

1-93 

69 

1.70 

.20 

21 

2.39 

I.76 

71 

8.J+7 

.005 

22 

1.76 

1.52 

72 

1.82 

.18 

23 

3.62 

3.66 

72 

.05 

.82 

2k 

3.27 

3.34 

72 

.16 

.69 

25 

3.00 

3.00 

70 

0.0 

1.00 

26 

3.00 

3.00 

72 

0.0 

1.00 

27 

3.H 

2.93 

72 

I.02 

.32 

28 

3.09 

2.71 

71 

3.00 

.09 

29 

2.89 

2.60 

k9 

1.23 

.27 

30 

3.kh 

3.^5 

72 

.00 

.98 

31 

3.11 

3.59 

72 

6.91 

.01 

32 

2.63 

3-00 

7 

.15 

.71 

33 

2.56 

3-00 

10 

1.05 

.33 

3^ 

2.60 

2.59 

72 

.00 

.95 

35 

2.76 

3.00 

72 

1.1*5 

.23 

36 

3.56 

3.56 

32 

.00 

.98 

37 

2.17 

2.29 

2k 

.08 

.78 

38 

3.27 

3-17 

72 

.k2 

.52 

39 

3-33 

3.68 

68 

5.81 

.02 

4o 

2.60 

2.97 

72 

2.84 

.10 

4i 

3.31 

3.07 

72 

1.45 

.23 

k2 

2.61 

2.55 

71 

.06 

.81 
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